Skip to content

feat: report failure diagnostics in relay test:fail events - #15

Merged
kevinccbsg merged 1 commit into
mainfrom
feature/failure-diagnostics
Sep 5, 2026
Merged

kevinccbsg merged 1 commit into
mainfrom
feature/failure-diagnostics

Conversation

@kevinccbsg

Copy link
Copy Markdown
Member

Surfaces the failure-diagnostics block from BRIKEV/twd#335 in relay runs, so a CLI failure names the route the test was on and which mock rules never fired.

  Failed tests (1):
    × Catalog > lists products
      ── TWD diagnostics ─────────────────────────────────────
      location    /cg-1/settings/catalog
      mock rules  6/7 triggered — catalog never requested
      ────────────────────────────────────────────────────────

      expected 0 to be 3

Why the relay showed nothing

createBrowserClient supplies its own events object to TestRunner, so the sidebar's onFail never runs under the relay — ours does, and it sent only error.message. The block now goes above the message, matching the sidebar and runner-ci: the accessible-roles dump lives inside err.message and would otherwise bury it.

src/cli/run.ts needed no change — it already indents multi-line errors.

The renderer is a port, not an import

NEXT-DIAGNOSTICS.md proposed await import('twd-js') to reach formatDiagnostics. I went the other way, for two reasons:

  1. Weight. twd-js's public entry is ~470 KB (sidebar, chai, theme) versus 226 bytes for twd-js/runner. That is a lot to pull into the page for a pure string formatter.
  2. It isn't exported. formatDiagnostics is still private on twd/main after #335 merged, and there is no light subpath for it. Importing it would need a second twd-js change and a second release.

The relay needs only the data, and that already arrives on the handler through twd-js/runner, which we import today. So src/browser/diagnostics.ts ports the ~25-line pure renderer, in the same spirit as the existing structural TwdHandler and VitePlugin interfaces.

The trade-off: the format is now duplicated and can drift. src/tests/browser/diagnostics.spec.ts mirrors twd-js's own spec case-for-case to make drift show up as a failure, and CLAUDE.md records that the port must be updated if twd-js's format changes. If you'd rather have a single source of truth, the alternative is exporting the formatter from a light twd-js subpath and importing it — happy to switch.

No peer-range bump

Nothing new is imported from twd-js, so >=1.4.0 stands. On a twd-js that doesn't report diagnostics the block is simply absent — same as for a test that failed one attempt then passed on retry (runner.ts clears diagnostics at the start of each attempt).

This is a safe no-op until you publish. Merging before the release changes nothing at runtime; the block starts appearing the moment the new twd-js is installed, with no further relay change.

Tests

  • diagnostics.spec.ts — 9 cases mirroring twd-js's spec: header/footer rule, location alone, summary when all triggered, single miss inline, several misses listed, cap at five plus +N more, and the no-diagnostics passthrough.
  • createBrowserClient.diagnostics.spec.ts — the first test that drives the real browser client against a real relay. The existing browser suite simulates a browser-role connection, so it could not catch the composition being removed from onFail. Verified red-green: reverting the one-line wiring fails this test. Side effect: createBrowserClient.ts coverage goes 0% → 49%.
  • run.spec.ts — one case proving the block survives to the terminal one line per row, correctly indented.

Verification

  • npm run test:ci → 96/96 passing, 11 files (was 85/9)
  • npx tsc --noEmit → clean
  • npm run build → OK; 3 rolled-up .d.ts, 0 stub files
  • attw --pack . → node16 (from ESM) green on all 3 entries
  • npm audit → 0 vulnerabilities
  • diagnostics.ts at 100% statement coverage

Left out deliberately

test.logs = [error.message] at line 214 is still an assignment, not an append — your call per NEXT-DIAGNOSTICS.md, and its only effect is in the twd-js sidebar, which nothing here can verify. Worth noting the window is narrower than the doc suggests: runner.ts already resets test.logs = [] at the start of every attempt, so only entries pushed during the test body are lost.

🤖 Generated with Claude Code

twd-js attaches a diagnostics snapshot to a failed test (BRIKEV/twd #335)
naming the route the test was on and which mock rules never fired. The
relay supplies its own `events` object to `TestRunner`, so the sidebar's
`onFail` never runs under the relay and that block was being dropped.

`onFail` now composes the block above the error message, matching the
sidebar and `runner-ci` — the accessible-roles dump lives inside
`err.message` and would otherwise bury it. `src/cli/run.ts` needed no
change; it already indents multi-line errors.

The renderer is a deliberate port of twd-js's internal
`src/utils/diagnostics.ts` rather than an import. twd-js keeps
`formatDiagnostics` private, and its only public entry pulls in the
sidebar, chai and the theme (~470 KB) — far too much to load into the
page for a pure string formatter. The relay needs only the data, which
already arrives on the handler via `twd-js/runner`. `diagnostics.spec.ts`
mirrors twd-js's own spec so the two formats stay byte-identical.

No peer-range bump: nothing new is imported from twd-js. On a version
that does not report diagnostics the block is simply absent, as it is
for a test that failed one attempt then passed on retry.

Adds the first integration test that drives the real browser client
against a real relay, verified red-green: removing the composition from
`onFail` fails it. That lifts createBrowserClient.ts coverage 0% -> 49%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kevinccbsg
kevinccbsg merged commit 6595bf6 into main Sep 5, 2026
5 checks passed
@kevinccbsg
kevinccbsg deleted the feature/failure-diagnostics branch September 5, 2026 23:24
kevinccbsg added a commit that referenced this pull request Sep 6, 2026
Minor, not patch: this release adds the failure-diagnostics block to
relay `test:fail` events (#15). Also carries the dependency bumps that
brought `npm audit` to zero (#14).

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant